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SCOPE OF CLAIMS 

1. A captured image data processing method characterized by: 

a step of subtracting, from a pixel value of each pixel which 

5 constitutes second image data acquired by way of an imaging process 
causing a first light source to emit light, a pixel value of a corresponding 
pixel of first image data acquired by way of an imaging process causing 
the first light source not to emit light, and obtaining difference image 
data corresponding to an image captured in an irradiation environment 

10 of only the first light source; 

a difference image data pixel value adjustment processing step of 
performing a pixel value adjustment process with respect to the 
difference image data; and 

a final pixel value adjustment image generation step of applying 

15 image data of adjusted pixel values generated in the difference image 
data pixel value adjustment processing step, and generating a final pixel 
value adjustment image. 

2 The captured image data processing method according to claim 1, 
20 characterized by further including : 

a first image data pixel value adjustment processing step of 
performing pixel value adjustment process with respect to the first image 
data; wherein 

the final pixel value adjustment image generation step is a step of 
25 performing a pixel value addition process for the corresponding pixels of 
two image data of adjusted pixel values acquired by the difference image 
data pixel value adjustment processing step and the first image data 
pixel value adjustment processing step, and generating final pixel value 
adjustment image data, 

30 

3. The captured image data processing method according to claim 2, 
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characterized in that the pixel value adjustment process is a white 
balance adjustment process; 

the difference image data pixel value adjustment processing step 
is a white balance adjustment process according to a parameter set based 
5 on an light component of the first light source, and the first image data 
pixel value adjustment processing step is a white balance adjustment 
process according to a parameter set based on an ambient light 
component which does not contain the first light source. 

10 4. The captured image data processing method according to claim 3, 
characterized in that the parameter is a parameter represented by a 3x3 
matrix, the matrix applied for conversion of color components which 
constitute a color of each pixel. 

15 5. The captured image data processing method according to claim 4, 
characterized in that the 3x3 matrix is a matrix set as 0 with exception to 
diagonal components. 

6. The captured image data processing method according to claim 1, 
20 characterized by further including a pixel value addition step of adding a 
pixel value of each pixel of the image data of adjusted pixel values 
generated in the difference image data pixel value adjustment processing 
step to a pixel value of a corresponding pixel of the first image data; 
wherein 

25 the final pixel value adjustment image generation step is a step of 

performing the pixel value adjustment with respect to the image data 
generated in the pixel value addition step, and generating the final pixel 
value adjustment image data. 

30 7. The captured image data processing method according to claim 6, 
characterized in that the pixel value adjustment process is a white 
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balance adjustment process; and 

the difference image data pixel value adjustment processing step 
is a white balance adjustment process according to a parameter set based 
on ambient light component which does not contain the first light source. 

5 

8. The captured image data processing method according to claim 1, 
c±iaracterized by further including: 

a motion detecting step of detecting a moving portion where a 
subject moves according to the difference data of a plurality of image 
10 data; and 

a moving portion pixel value adjustment step of performing pixel 
value adjustment process with respect to the moving portion. 

9. The captured image data processing method according to claim 8, 
15 characterized in that the moving portion pixel value adjustment step has: 

a step of calculating a light intensity scalar ratio of two light 
sources when the first light source is caused or not caused to emit light, 
as data corresponding to a pixel of a motionless portion adjacent to a 
moving portion; 

20 a step of using a radial basis function (RBF) so as to calculate the 

light intensity scalar ratio corresponding to each pixel of the moving 
portion; 

a first pixel value calculation step of calculating the pixel value, as 
the first pixel value, of each pixel of the moving portion in an image 
25 which is equivalent to the image captured in an irradiation environment 
of the first light source only, based on the light intensity scalar ratio 
corresponding to each pixel of the moving portion; 

a second pixel value calculation step of calculating the pixel value, 
as the second pixel value, of each pixel of the moving portion in an image 
30 which is equivalent to the image captured in an ambient light irradiation 
environment where the first light source is not included, based on the 
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light intensity scalar ratio corresponding to each pixel of the moving 
portion; 

a first pixel value adjustment processing step of performing the 
pixel value adjustment process, based on the first pixel value calculated 
5 in the first pixel value calculation step; 

a second pixel value adjustment processing step of performing 
the pixel value adjustment process, based on the second pixel value 
calculated in the second pixel value calculation step; and 

a pixel value addition step of adding two adjustment pixel values 
10 generated in the first pixel value adjustment processing step and the 
second pixel value adjustment processing step. 

10, The captured image data processing method according to claim 9, 
characterized in that the pixel value adjustment process is a white 

15 balance adjustment process; 

the first pixel value adjustment processing step is a white balance 
adjustment process according to a parameter set based on light 
component of the first light source; and 

the second pixel value adjustment processing step is a white 
20 balance adjustment process according to a parameter set based on an 
ambient light component which does not contain the first light source. 

11. The captured image data processing method according to claim 8, 
characterized in that the moving portion pixel value adjustment step 

25 further has: 

a step of calculating a color component ratio (a*, Og, a*,) of a pixel 

value of an inner peripheral pixel of the moving portion to an average 

pixel value of outer peripheral pixels of the moving portion adjacent to 

the inner peripheral pixel of the moving portion; 
30 a step of constructing the radial basis function (RBF) based on the 

color component ratio corresponding to each pixel by considering all the 
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inner peripheral pixels in the moving portion as sample points; and 

a step of calculating the color component ratio corresponding to 
each pixel of the moving portion, based on the radial basis function (RBF), 
and multiplying the color component ratio with an image to be 
5 compensated and obtaining a compensation pixel value. 

12. The captured image data processing method according to claim 1, 
characterized by further including: 

a motion detecting step of detecting a moving portion where a 
10 subject moves according to the difference data of a plurality of image 
data; 

and a step of performing the pixel value adjustment process 
based on the second image data acquired by way of an imaging process 
causing the first light source to emit light when a proportion of the whole 
15 image of the moving portion is higher than a predetermined threshold 
value; wherein 

the pixel value adjustment data based on the second image data 
is set as a final pixel value adjustment data. 

20 13. The captured image data processing method according to claim 
12, characterized in that the pixel value adjustment process based on the 
second image data is either a white balance adjustment process 
according to a parameter set based on a light component of the first light 
source, the white balance adjustment process according to a parameter 

25 set based on ambient light component not containing the first light 
source, or a white balance adjustment process according to a parameter 
set based on an intermediate light component between the first light 
source and ambient ligjht component not containing the light component 
of the first light source. 

30 

14. A captured image data processing apparatus characterized by: 
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a memory for storing first image data acquired by way of an 
imaging process causing the first light source not to emit light, and a 
second image data acquired by way of an imaging process causing the 
first light source to emit light; and 
5 a data processing unit for performing a pixel value adjustment 

process based on the image data stored in the memory; the data 
processing unit including 

a difference image data acquisition unit for subtracting, 
from a pixel value of each pixel which constitutes the second 
10 image data, a pixel value of a corresponding pixel of the first 

image data, so as to acquire difference image data corresponding 
to image captured in an irradiation environment of only the first 
light source; 

a difference image data pixel value adjustment unit for 
15 performing the pixel value adjustment process with respect to 

the difference image data; and 

a final pixel value adjustment unit for performing a final 
pixel value adjustment by using an image data of adjusted pixel 
values generated in the difference image data pixel value 
20 adjustment unit 

15. The captured image data processing apparatus according to 
claim 14, characterized in that the data processing unit further includes 
first image data pixel value adjustment unit for performing pixel value 

25 adjustment process with respect to the first image data; wherein 

the final pixel value adjustment unit is arranged to perform pixel 
value addition process for corresponding pixels of two image data of 
adjusted pixel values generated in the difference image data pixel value 
adjustment unit and the first image data pixel value adjustment unit, and 

30 generate the final pixel value adjustment image data 
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16. The captured image data processing apparatus according to 
claim 15, characterized in that the pixel value adjustment process is a 
white balance adjustment process; 

the difference image data pixel value adjustment unit is arranged 
5 to perform the white balance adjustment process according to a 
parameter set based on a light component of the first light source; and 

the first image data pixel value adjustment unit is arranged to 
perform the white balance adjustment process according to a parameter 
set based on ambient light component not including the first light source. 

10 

17. The captured image data processing apparatus according to 
claim 14, characterized in that the data processing unit further includes a 
pixel value adding unit for adding a pixel value of each pixel of the 
image data of adjusted pixel values generated in the difference image 

15 data pixel value adjustment unit to a pixel value of a corresponding pixel 
of the first image data, and the final pixel value adjustment image 
generating unit performs pixel value adjustment with respect to image 
data generated in the pixel value adding unit, and generate the final pixel 
value adjustment image data. 

20 

18. The captured image data processing apparatus according to 
claim 17, characterized in that the pixel value adjustment process is a 
white balance adjustment process; and 

the difference image data pixel value adjustment unit performs 
25 white balance adjustment process according to a parameter set based on 
ambient light component not including the first light source. 

19. The captured image data processing apparatus according to 
claim 14, characterized in that the data processing unit includes : 

30 a motion detection unit for detecting a moving portion of a 

subject based on difference data between a plurality of image data; and 
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a moving portion pixel value adjustment unit for performing 
pixel value adjustment process with respect to the moving portion. 

20 The captured image data processing apparatus according to 
5 claim 19, wherein the moving portion pixel value adjustment unit is 

characterized by: 

calculating a light intensity scalar ratio of two light sources when 

the first light source is caused or not caused to emit light, as data 

corresponding to a pixel of a motionless portion adjacent to a moving 
10 portion; 

calculating light intensity scalar ratio corresponding to each pixel 
of the moving portion by applying a radial basis function (RBF); 

calculating pixel value as first pixel value of each pixel of the 
moving portion in an image which is equivalent to the captured image in 
15 an irradiation environment of only the first light source, based on light 
intensity scalar ratio corresponding to each pixel of the moving portion; 

calculating pixel value as second pixel value of each pixel of the 
moving portion in an image which is equivalent to the captured image in 
an ambient light irradiation environment where the first light source is 
20 not included, based on light intensity scalar ratio corresponding to each 
pixel of the moving portion; 

performing pixel value adjustment process, based on the first 
pixel value; -* 

performing pixel value adjustment process based on the second 
25 pixel value; and 

adding the generated two adjustment pixel values to perform a 
moving portion pixel value adjustment process. 

21. The captured image data processing apparatus according to 
30 claim 20, dharacterized in that the pixel value adjustment process is a 
white balance adjustment process; 
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the first pixel value adjustment process is performed as the white 
balance adjustment process according to a parameter set based on a light 
component of the first light source; and 

the second pixel value adjustment process is performed as a 
5 white balance adjustment process according to a parameter set based on 
ambient light component not including the first light source. 

22. The captured image data processing apparatus according to 
claim 19, wherein the moving portion pixel value adjustment unit is 

10 characterized by: 

calculating a color component ratio (a*, Og, otb) of the pixel value 
of an inner peripheral pixel of the moving portion to an average value of 
the pixel value of an outer peripheral pixel of the moving portion 
adjacent to the inner peripheral pixel of the moving portion; 

15 constructing a radial basis function (RBF) based on the color 

component ratio coiresponding to each pixel by considering all the inner 
peripheral pixels in the moving portion as sample points; 

calculating the color component ratio corresponding to each pixel 
of the moving portion based on the radial basis function (RBF); and 

20 multiplying the color component ratio with an image to be 

compensated, and obtaining a compensation pixel value. 

23. The captured image data processing apparatus according to 
claim 19, wherein the data processing unit is characterized by: 

25 performing the pixel value adjustment process based on the 

second image data acquired by way of an imaging process causing the 
first light source to emit light when a proportion of the whole image of 
the moving portion detected by the motion detection unit is higher than 
a predetermined threshold value, and the pixel value adjustment data 

30 based on the second image data is set as final pixel value adjustment 
data. 



92 



S03P0865 



24. The captured image data processing apparatus according to 
claim 23, characterized in that the pixel value adjustment process based 
on the second image data performs either a white balance adjustment 

5 process according to the parameter set up based on the light component 
of the first light source, the white balance adjustment process according 
to the parameter set up based on ambient light component not including 
the first light source, or the white balance adjustment process according 
to the parameter set up based on an intermediate light component 

10 between the light component of the first light source and ambient light 
component not including the first light source. 

25. A captured image data processing method characterized by: 

a step of acquiring first image data by way of an imaging process 
15 causing the first light source not to emit light; 

a step of acquiring a second image data by way of an imaging 
process causing the first light source to emit light; 

a step of subtracting, from a pixel value of each pixel which 
constitutes the second image data, a pixel value of a corresponding pixel 
20 of the first image data, and acquiring a difference image data 
corresponding to the image captured in an irradiation environment of 
only the first light source; 

a difference image data pixel value adjustment processing step of 
performing the pixel value adjustment process with respect to the 
25 difference image data; and 

a final pixel value adjustment image generation step of 
generating a final pixel value adjustment image by using the adjusted 
pixel value image data generated in the difference image data pixel value 
adjustment processing step. 

30 

26. A captured image data processing method characterized by: 
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a step of acquiring a plurality of image data captured at mutually 
different times in a situation where Hght-emitting conditions of a first 
light source are the same; 

a step of acquiring motion information based on comparison 
5 among the plurality of image data; and 

a step of performing a process according to white balance set 
based on the motion information, with respect to either the plurality of 
image data, a portion thereof, all of them, or other image data. 

10 27. The captured image data processing method according to claim 
26, characterized in that, in the step of performing the process according 
to the white balance set based on the motion information, the white 
balance is adjusted with respect to a portion corresponding to other 
image data captured either during a period when the plurality of image 
15 data is captured, or immediately before or immediately after the plurality 
of image data is captured,. 

28. The captured image data processing method according to claim 
26, characterized in that, when it is determined that a motion is large 
based on the motion information, the white balance adjustment process 
is carried out based on either ambient light, flash light, or a virtual light 
source between ambient light and the flash light, in the step of 
performing the process according to the white balance set up based on 
the motion information 

29. The captured image data processing method according to claim 
26, characterized in that the step of performing the white balance 
treatment process set based on the motion information further includes a 
step of adaptively switching the white balance adjustment processes for 
each portion of the image, instead of an uniform process for the entire 
image, and, based on the motion information, the image is interpolated 
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from the image data of a still portion adjacent to a corresponding portion 
so as to obtain image data of the corresponding portion. 

30. A captured image data processing apparatus characterized by: 

5 an imaging means for acquiring a plurality of image data by way 

of an imaging process causing a first light source not to emit light and an 
imaging process causing the first light source to emit light; 

a memory for storing first image data acquired by way of the 
imaging process causing the first light source not to emit light and a 
10 second image data acquired by way of the imaging process causing the 
first light source to emit light; and 

a data processing unit for performing a pixel value adjustment 
process based on the image data stored in the memory; wherein 
the data processing unit includes: 
15 a difference image data acquisition unit for subtracting, from a 

pixel value of each pixel which constitutes the second image data, a pixel 
value of a corresponding pixel of the first image data, so as to acquire a 
difference image data corresponding to the image captured in an 
irradiation environment of only the first light source; 
20 a difference image data pixel value adjustment unit for 

performing a pixel value adjustment process with respect to the 
difference image data; and 

a final pixel value adjustment unit for performing a final pixel 
value adjustment by using an image data of adjusted pixel values 
25 generated in the difference image data pixel value adjustment unit 

31. A captured image data processing apparatus characterized by: 
means for acquiring a plurality of image data captured at 

mutually different times in a situation where light-emitting and imaging 
30 conditions of a first light source are the same; 

means for acquiring motion information based on a comparison 
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between the plurality of image data; and 

means for performing a process according to a white balance set 
based on the motion information, with respect to either the plurality of 
image data, a part thereof, all of them, or other image data. 

5 

32 A computer-readable program for making a computer execute 

steps of image data processing, the computer program characterized by: 
a step of acquiring first image data by way of an imaging process 

causing a first light source not to emit light; 
10 a step of acquiring a second image data by way of an imaging 

process causing the first light source to emit light 

a step of subtracting, from a pixel value of each pixel which 

constitutes the second image data, a pixel value of a corresponding pixel 

of the first image data, and acquire a difference image data 
15 corresponding to the image captured in an irradiation environment of 

only the first light source; 

a difference image data pixel value adjustment processing step of 

performing a pixel value adjustment process with respect to the 

difference image data; and 
20 a final pixel value adjustment image generation step of 

generating a final pixel value adjustment image by using the adjusted 

pixel value image data generated in the difference image data pixel value 

adjustment processing step. 

25 33. An image data processing method for performing a pixel value 

compensation process with respect to image data having a saturation 

pixel value, the method characterized by: 

a temporary compensation pixel value calculation step of 

inputting a plurality of image data including first image data to be 
30 compensated having saturation pixels in which at least a part of pixels of 

the image data are set up as a maximum pixel value, and one or more 



S03P0865 



sub-image data captured under an exposure condition different from 
that for the first image data; acquiring an exposure condition difference 
data, based on mutually corresponding pixel value differences in a 
plurality of input image data, which indicates a difference in the 
5 exposure condition at a time of acquiring each image data; and 
calculating a temporary compensation pixel value of the saturation pixel 
in the first image data, based on the exposure condition difference data; 
and 

a compensation pixel value calculation step of performing a 
10 normalization process for correcting the temporary compensation pixel 
value to a pixel value within an outputable range. 

34. The image data processing method according to claim 33, 
characterized in that 

15 the first image data has pixel values (Rcl, Gel, Bel) in saturation 

pixel position (Xc, Yc), and has pixel values (Rul, Gul, Bui) in 
non-saturated pixel (Xu, Yu); the sub-image data includes a second 
image data as a captured image of a lower intensity exposure than the 
first image data and a third image data as a captured image of an 

20 exposure of still lower intensity than the second image data; 

the temporary compensation pixel value calculation step 
performs a calculation process according to the following equations, 
assuming that the pixel value in the position corresponding to the 
non-saturated pixel (Xu, Yu) of the first image data in the second image 

25 data and the third image data are (Ru2, Gu2, Bu2) and (Ru3, Gu3, Bu3), 



P(Xu 9 Yu) = 
.... (equation 1) 



( (Rul - Ru3) V ( (Gul - Gu3) } 2 ( (Bui - Bu3) 



\(Ru2-Ru3)J \(Gu2-Gu3)J K(Bu2-Bu3)J 
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n 

.... (equation 2) 

and calculates an exposure condition difference data :S based on 
the above (equation 1) and (equation 2). 

5 35. The image data processing method according to claim 33, 
characterized in that 

the compensation pixel value calculation step performs a 
calculation process according to the following equations, assuming that 
the pixel values of the position corresponding to the saturation pixel (Xc, 
10 Yc) of the first image data in the second image data and the third image 
data are (Rc2, Gc2, Bc2) and (Rc3, Gc3, Bc3), 
Rcq = (Rc2 - Rc3) x S + Rc3 
Gcq=(Gc2-Gc3)xS + Gc3 
Bcq = (Bc2 - Bc3) x S + Bc3 
15 .... (equation 3) 

and, based on (equation 3), calculates the temporary 
compensation pixel values (Rcq, Gcq, Bcq) of the saturation pixel of the 
first image data. 

20 36. The image data processing method according to claim 33, 

characterized in that 

the compensation pixel value calculation step calculates a 

maximum value: Dmax of all data of pixel values (Ru, Gu, Bu) of the 

non-saturated pixel (Xu, Yu) of the first image, and the temporary 
25 compensation pixel values (Rcq, Gcq, Bcq) with respect to the saturation 

pixel (Xc, Yc), performing the calculation process according to the 

following equations, 
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where the non-saturated pixel (Xu, Yu), 
Ruf = Ru/Dmax 
Guf - Gu/Dmax 
Buf = Bu/Dmax 

5 

where the saturation pixel (Xc, Yc), 
Rcf = Rcq/Dmax 
Gcf = Gcq/Dmax 
Bcf = Bcq/Dmax 
10 .... (equations 4) 

and, based on (equations 4), calculates the compensation pixel 
values (Ruf, Guf, Buf) of the non-saturated pixel of the first image data 
and the compensation pixel values (Rdf, Gcf, Bcf) of the saturation pixel. 

15 

37. The image data processing method according to claim 33, 
characterized in that the difference in exposure conditions between the 
first image data and one or more sub-image data is a difference in 
irradiation light intensity; and the temporary compensation pixel value 

20 calculation step calculates a ratio of amount of irradiation light based on 
the difference in the irradiation light intensity of a plurality of image data 
as the exposure condition difference data. 

38. The image data processing method according to claim 33, 
25 characterized in that the difference in the exposure conditions between 

the first image data and one or more sub-image data is the difference in 
exposure time, and the temporary compensation pixel value calculation 
step calculates the ratio of irradiation light intensity based on the 
difference in the exposure time of a plurality of image data as the 
30 exposure condition difference data. 
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39. The image data processing method according to claim 33, 
characterized in that the temporary compensation pixel value calculation 
step and the compensation pixel value calculation step are characterized 
by calculating a compensation data on each signal component of a color 

5 image in the first image, 

40. The image data processing method according to claim 33, 
characterized in that the temporary compensation pixel value calculation 
step and the compensation pixel value calculation step calculate the 

10 compensation data according to a luminance component of the first 
image. 

41. An imaging method for capturing image data to be stored in a 
memory unit, the method characterized by: 

15 an imaging step of capturing image upon setting up different 

exposure conditions; 

a temporary compensation pixel value calculation step of 
inputting a plurality of image data including first image data to be 
compensated and having a saturation pixel which at least part of pixels 

20 in the image data is set up as a maximum pixel value, and one or more 
sub-image data captured under an exposure condition different from 
that for the first image data; acquiring an exposure condition difference 
data, based on mutually corresponding pixel value differences in a 
plurality of input image data, which indicates a difference in the 

25 exposure condition at a time of acquiring each image data, and 
calculating a temporary compensation pixel value of the saturation pixel 
in the first image data, based on the exposure condition difference data; 

a compensation pixel value calculation step of performing a 
normalization process of correcting the temporary compensation pixel 

30 value to a pixel value within an outputable range; and 

a store step of storing the image data constituted by the pixel 



100 



S03P0865 



value compensated by the compensation pixel value calculation step. 

42 The imaging method according to claim 41, characterized in that 
the difference between the exposure conditions is either a difference in 
5 irradiation light intensity, or a difference in exposure time, the imaging 
step captures a plurality of image data under the different setup 
conditions of the irradiation light intensity or the exposure time, and the 
temporary compensation pixel value calculation step calculates the ratio 
of irradiation light intensity based on the difference in the irradiation 
10 light intensity or the exposure time of the plurality of image data as the 
exposure condition difference data. 

43. A captured image data processing apparatus for performing a 
pixel value compensation process with respect to image data having a 

15 saturation pixel value, the image data processing apparatus 
characterized by: 

a temporary compensation pixel value calculation means for 
inputting a plurality of image data including first image data to be 
compensated and having saturation pixels in which at least a part of 

20 pixels of the image data are set up as a maximum pixel value, and one or 
more sub-image data captured under an exposure condition different 
from that for the first image data; acquiring an exposure condition 
difference data, based on mutually corresponding pixel value differences 
in a plurality of input image data, which indicates a difference in the 

25 exposure condition at a time of acquiring each image data, and 
calculating a temporary compensation pixel value of the saturation pixel 
in the first image data, based on the exposure condition difference data; 
and 

compensation pixel value calculation means for performing a 
30 normalization process for correcting the temporary compensation pixel 
value to a pixel value within an outputable range. 
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44. A computer program for executing an image data process 
including a pixel value compensation process with respect to an image 
data which has a saturation pixel value, the computer program 
5 characterized by: 

a temporary compensation pixel value calculation step of 
inputting a plurality of image data including first image data to be 
compensated and having a saturation pixels in which at least part of 
pixels in the image data are set up as a maximum pixel value, and one or 

10 more sub-image data captured under an exposure condition different 
from that for the first image data; acquiring an exposure condition 
difference data, based on corresponding pixel value differences in a 
plurality of input image data, which indicates a difference in the 
exposure condition at a time of acquiring each image data, and 

15 calculating a temporary compensation pixel value of the saturation pixel 
in the first image data, based on the exposure condition difference data; 
and 

a compensation pixel value calculation step of performing a 
normalization process for correcting the temporary compensation pixel 
20 value to a pixel value within an outputable range. 
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